Method for generating and displaying panorama images based on rendering engine and a display apparatus

ABSTRACT

The present application discloses method for generating a panorama image based on a rendering engine associated with a display apparatus. The method includes determining a display model configured to be a polygon prism for a rendering engine corresponding to a panorama view of a scene. Additionally, the method includes using multiple sampling cameras associated with the rendering engine to capture multiple sample sub-images of the panorama view in respective directions towards multiple sub-planes of the polygon prism. The method further includes attaching the multiple sample sub-images to respective multiple sub-planes to form a constructed display model. Furthermore, the method includes rendering the constructed display model by the rendering engine to reconstruct a panorama image and displaying the panorama image on the display apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a national stage application under 35 U.S.C. § 371of International Application No. PCT/CN2018/116738, filed Nov. 21, 2018,the contents of which are incorporated by reference in the entirety.

TECHNICAL FIELD

The present invention relates to display technology, more particularly,to a method for generating panorama images based on rendering engine,and a display apparatus implementing the method.

BACKGROUND

In some applications, mobile virtual reality (VR) apparatus needs todisplay a panorama image instead of rendering real scene in real-time tosmoothly play the VR video. A conventional panorama image includes asingle 360° image generated from a texture model in a sphere shape.Several issues existed with this sphere model include 1) display imagedistortion caused by the patch aggregation of the single panorama imagein vertical direction from a top point to a bottom point of the spheremodel; 2) limitation of maximum image resolution of an image in acertain field-of-view angle due to lack of decoding ability in image'spixel resolution of the single panorama image under the samefield-of-view angle.

SUMMARY

In an aspect, the present disclosure provides a method for generating apanorama image based on a rendering engine associated with a displayapparatus. The method includes determining a display model configured tobe a polygon prism for the rendering engine corresponding to a panoramaview of a scene. The method further includes using multiple samplingcameras associated with the rendering engine to capture multiple samplesub-images of the panorama view in respective directions towardsmultiple sub-planes of the polygon prism. Additionally, the methodincludes attaching the multiple sample sub-images to respective multiplesub-planes to form a constructed display model. Furthermore, the methodincludes rendering the constructed display model by the rendering engineto reconstruct a panorama image. Moreover, the method includesdisplaying the panorama image on the display apparatus.

Optionally, the polygon prism is an equilateral N-side polygon prism. Nis an integer no smaller than 3.

Optionally, the step of determining a display model includes determiningthe multiple sub-planes divided respectively from each side or top orbottom plane of the equilateral N-side polygon prism.

Optionally, the step of determining a display model further includesdetermining different image resolutions corresponding to respectivemultiple sub-planes of different planes of the equilateral N-sidepolygon prism and determining a width and a length as well as awidth-to-length ratio of each of the multiple sub-planes based on aratio of tan(u/2)/tan(v/2). u is a horizontal field-of-view angle and vis a vertical field-of-view angle projected from at least one of themultiple sampling cameras to each of the multiple sub-planes.

Optionally, the at least one of the multiple sampling cameras is locatedat a center of the display model.

Optionally, the step of using multiple sampling, cameras associated withthe rendering engine to capture multiple sample sub-images includesseparately sampling each of the multiple sample sub-images with anindependently-defined image resolution depending on a scene inrespective one direction toward one of multiple sub-planes projectedfrom one of the multiple sampling cameras.

Optionally, the independently-defined image resolution for a singlesub-plane is configured to be several times greater than a maximum imageresolution allowed for a single field-of-view image in a conventionalmodel.

Optionally, the step of using multiple sampling cameras associated withthe rendering engine to capture multiple sample sub-images includescapturing at least two sample sub-images in projected squaresrespectively for top view toward a top plane and bottom view toward abottom plane by slightly enlarging field-of-view angles to make theprojected squares to be larger than the top plane or the bottom plane.

Optionally, the step of using multiple sampling cameras associated withthe rendering engine to capture multiple sample sub-images includesusing two sets of sampling cameras to capture two sets of samplesub-images with parallax for the multiple sub-planes and reconstructinga 3D panorama image based on the two sets of sample sub-images withparallax.

Optionally, the step of attaching the multiple sample sub-images torespective multiple sub-planes includes performing UV mapping to addimage textures of each sample sub-image to a corresponding sub-plane andgenerating a constructed display model for the rendering engine.

Optionally, the step of using multiple sampling cameras associated withthe rendering engine to capture multiple sample sub-images furtherincludes sampling a series of sample sub-images time-sequentially with asampling frequency for each of the multiple sub-planes of the displaymodel; encoding all sample sub-images of a same sub-plane according toan order of being sampled to generate a sample sub-video; and attachingthe sample sub-video to a corresponding sub-plane to form theconstructed display model.

Optionally, the step of rendering the constructed display model includesrendering multiple sample sub-images respectively for the multiplesub-planes that are sampled at a same time to generate one panoramaimage at the same time and further generating a panorama video byencoding a series of panorama images sequentially in time.

Optionally, the step of displaying the panorama image comprisesdisplaying the panorama video on the display apparatus by at leastdisplaying a sub-video separately for respective one sub-plane in adirection of a field-of-view.

In another aspect, the present disclosure provides an apparatus forgenerating a panorama image based on a rendering engine associated witha display apparatus. The apparatus includes a memory and one or moreprocessors. The memory and the one or more processors are connected witheach other. The memory stores computer-executable instructions forcontrolling the one or more processors to determine a display modelconfigured to be a polygon prism for the rendering engine correspondingto a panorama view of a scene; to use multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images of the panorama view in respective directions towardsmultiple sub-regions of the polygon prism; to attach the multiple samplesub-images to respective multiple sub-regions to form a constructeddisplay model; to render the constructed display model by the renderingengine to reconstruct a panorama image; and to display the panoramaimage on the display apparatus.

Optionally, the one or more processors is controlled by a programmableinstruction to create a display model configured as an equilateralpolygon prism with each of side, top, and bottom planes being, divided,into one or more sub-planes. Each sub-plane is independently definedwith an image resolution.

Optionally, the one or more processors includes a rendering enginehaving multiple sampling cameras configured to capture multiple samplesub-images of a scene in multiple projection directions from at leastone of the multiple sampling cameras located inside the display modeltowards one of the respective multiple sub-regions.

Optionally, the one or more processors is controlled by a programmableinstruction to attach each of the multiple sample sub-images as texturesto respective one of the multiple sub-regions of the polygon prism toform a constructed display model for the rendering engine.

Optionally, the rendering engine is configured to render the multiplesample sub-images associated with the respective multiple sub-regions ofthe constructed display model to generate a panorama image.

Optionally, the rendering engine is configured to sequentially rendermultiple sets of the multiple sample sub-images associated with therespective multiple sub-regions of the constructed display model, eachset of the multiple sample sub-images being sequentially captured by arespective one of the multiple sampling cameras with a samplingfrequency and encoded in a timing order to generate respective multiplesample sub-videos associated with the multiple sub-regions, and togenerate a panorama video.

In yet another aspect, the present disclosure provides a displayapparatus including a display panel coupled to the apparatus describedherein for generating a panorama image or panorama video and beingconfigured to display a panorama image or panorama video as a virtualreality display.

In still another aspect, the present disclosure provides acomputer-program product including a non-transitory tangiblecomputer-readable medium having computer-readable instructions thereon.The computer-readable instructions is executable by a processor to causethe processor to perform determining a display model configured to be apolygon prism for a rendering engine corresponding to a panorama view ofa scene; using multiple sampling cameras associated with the renderingengine to capture multiple sample sub-images of the panorama view inrespective directions towards multiple sub-regions of the polygon prism;attaching the multiple sample sub-images to respective multiplesub-regions to form a constructed display model; and rendering theconstructed display model by the rendering engine to reconstruct apanorama image.

BRIEF DESCRIPTION OF THE FIGURES

The following drawings are merely examples for illustrative purposesaccording to various disclosed embodiments and are not intended to limitthe scope of the present invention.

FIG. 1 is a flow chart showing a method for generating a panorama imagebased on a rendering engine according to some embodiments of the presentdisclosure.

FIG. 2 is a schematic diagram comparing a conventional model and adisplay model for a rendering engine according to an embodiment of thepresent disclosure.

FIG. 3 is a schematic diagram showing a sample sub-image of a scenesampled by a sampling camera based on the display model according to anembodiment of the present disclosure.

FIG. 4 is an exemplary diagram showing eight sample sub-images sampledfrom eight planes of a hexagon prism display model according to anembodiment of the present disclosure.

FIG. 5 is a schematic diagram of a virtual panorama image generated froma constructed display model by running a rendering engine according toan embodiment of the present disclosure.

FIG. 6 is an exemplary diagram showing visual effects from severalrandom field-of-view angles out of the virtual panorama image generatedby the rendering engine according to some embodiments of the presentdisclosure.

FIG. 7 is a schematic diagram of a display model for the renderingengine for generating a panorama image according to another embodimentof the present disclosure.

DETAILED DESCRIPTION

The disclosure will now be described more specifically with reference tothe following embodiments. It is to be noted that the followingdescriptions of some embodiments are presented herein for purpose ofillustration and description only. It is not intended to be exhaustiveor to be limited to the precise form disclosed.

For a VR display apparatus having built-in high image resolution, it isdesirable that the content images to be displayed thereof are alsoprovided in comparable high resolution. One source of thehigh-resolution content images is to sample multiple high-resolutionimages directly from real scenes by multiple sampling cameras based on aproper display model for a rendering engine associated with the displayapparatus.

Accordingly, the present disclosure provides, inter alia, a method forgenerating a panorama image based on a rendering engine in a displayapparatus, an apparatus for generating a panorama image, and a displayapparatus having the same that substantially obviate one or more of theproblems due to limitations and disadvantages of the related art.

In one aspect, the present disclosure provides a method for generating apanorama image based on a rendering engine in a display apparatus. FIG.1 is a flow chart showing a method for generating a panorama image basedon a rendering engine according to some embodiments of the presentdisclosure. Referring to FIG. 1 , the method includes a step ofdetermining a display model configured to be a polygon prism for arendering engine corresponding to a panorama view of a scene.Optionally, the rendering engine is associated with a display apparatus.Optionally, the display apparatus is a virtual reality (VR) displayapparatus capable of displaying images with high resolution such as 4Kor 8K although it is not a limiting factor for the claims herein.Optionally, the rendering engine is a 3D rendering engine configured totreating image data for 3D image display. Optionally, the 3D renderingengine needs a 3D display model to be based upon.

FIG. 2 a schematic diagram comparing a conventional model and a displaymodel for a rendering engine according to an embodiment of the presentdisclosure. Conventionally, a sphere model 00 is used as a 3D displaymodel for generating a single 360° panorama image. It is relativelycomplex to use the sphere model 00 to generate the panorama imageespecially for the top point and the bottom point of the image whichoften induces patch aggregation to cause display distortion. But for aviewer using a virtual reality display apparatus to watch the panoramaimage, it is more likely the user would focus on the objects located inhorizontal view directions instead of the objects located in viewdirections of straight above or down, making information in the straightabove portion or down portion of the image much less important. In thisdisclosure, a novel display model such as a polygon prism model 10 asshown in FIG. 2 is provided. Optionally, the polygon prism model 10 isan equilateral polygon prism with N rectangular side planes and twoequilateral polygon planes respectively at top and bottom. For example,a first side plane is denoted as 101-1, a second side plane is denotedas 101-2, and a last one side plane is denoted as 101-N. A top plane isdenoted as 102. A bottom plane is denoted as 103. When the polygon prismmodel is in use for the rendering engine to generate a panorama image,it is required to attach at least N+2 image maps to respective N+2planes of the polygon prism model 10, compared with one single image mapattached to the spherical surface of the conventional model 00. Thedisplay model according to the present disclosure obviates the need ofperforming spherical transformation to a 2D image data and the need ofperforming edge stitching of the image, substantially simplifying theimage generation process.

Referring to FIG. 1 again, the method for generating a panorama imagebased on a rendering engine further includes using multiple samplingcameras associated with the rendering engine to capture multiple samplesub-images of the panorama view in respective directions towardsmultiple planes of the polygon prism. FIG. 3 is a schematic diagramshowing a sample sub-image of a scene sampled by a sampling camera basedon the display model according to an embodiment of the presentdisclosure. Referring to FIG. 3 , part B refers to a sampling camera inthe rendering engine being placed in a middle location of a scene andbeing configured to aim in a projection direction to sample an image ofa certain width and length. Part A1 of FIG. 3 further schematically laysthe sampling camera in terms of a rectangular lens matrix and aprojection rectangle based on a field-of-view (FOV) angle. Inparticular, a horizontal FOV angle u is defined as an angle measuredfrom the camera point O to a middle point of left boundary of theprojection rectangle from the lens matrix and another middle point ofright boundary of the projection rectangle from the lens matrix. Thehorizontal FOV angle u is effectively limited by N numbers of sideplanes of the polygon prism model. For example, N=6 for a hexagon prism,the FOV angle u is no bigger than 360°)/N=60° as each side plane is atmost just the projection rectangle in corresponding FOV angle. Part A2of FIG. 3 similarly shows a vertical FONT angle v defined as an anglemeasured from the camera point O to a middle point of top boundary ofthe lens matrix and another middle point of a bottom boundary of thelens matrix. This vertical FOV angle v is typically set to 90° in mostVR display apparatus. Part C of FIG. 3 shows an exemplary sampledsub-image in the FOV angles (u, v) in a certain projection directiontowards one side plane of the polygon prism model. After obtaining thehorizontal and vertical FOV angles, a projection rectangle used foradding textures is given a relative size characterized by awidth-to-height (w/h) ratio that satisfies the following relationship:w/h=tan(u/2)/tan(v/2).

The actual size of the projection rectangle is substantially matchedwith the side plane if the FOV angle is the u and the sampling camera islocated at the center in the hexagon prism model. Optionally, multiplesampling cameras are respectively employed to capture multiple samplesub-images of a panorama view in respective directions towards multipleplanes of the polygon prism model. Each captured sample sub-image issaved in a memory associated with the rendering engine.

In a VR display apparatus, typically each single image generated by arendering engine for a panorama image viewer has a limited maximum imageresolution, e.g., 8192×4096. So a maximum resolution of a conventionaldisplayable panorama image generated by the rendering engine is8192×4096 under a width-to-height ratio of 2:1. Any panorama image witha resolution higher than that will be compressed down to 8192×4096.Additionally, any partial image of the panorama image corresponding to aspecific FOV angle will be limited to an even smaller resolution. Forexample, if a viewer is in a middle of a model, a partial image within ahorizontal FOV angle of 60° and a vertical FOV angle of 90° is limitedto a resolution of (8192/6)×(4096/2)=1365×2048. In the polygon prismmodel 10, instead of a single image being mapped to the sphericalsurface, multiple sub-images are mapped respectively to multiple planesassociated with the polygon prism model 10. Unlike the limitation ofimage resolution for the single panorama image, each single sub-image inthe polygon prism model can have a resolution up to 8192×4096. Assumingthe polygon prism is a hexagon prism, there are six side planes in a360° view angle. Each sub-image attached to each side plane with maximumresolution of 8192×4096 can have a resolution of 24 times of theconventional partial image with maximum resolution of 1365×2048 withinthe same FOV angles. As each sub-image referred under the new polygonprism model is a displayable image, the image resolution displayed onthe VR display apparatus is substantially increased by 3 times or moreeven when each sub-image resolution is lowered from the above-mentionedmaximum resolution of 8192×4096 to 2865×4096. In an alternativeembodiment, if a display model provided with the rendering engineincludes additional sub-planes along a vertical direction, again eachsub-image in those vertically divided sub-planes of the display modelcan be captured in high resolution which will contribute a finalpanorama image with a substantially enhanced image resolution.

Referring to FIG. 1 , the method additionally includes a step ofattaching the multiple sample sub-images to respective multiple planesto form a constructed display model. Each sample sub-image captured fromthe scene by the sampling camera in the projection direction towards oneof the multiple planes is attached to the corresponding side or topplane. Optionally, the captured sample sub-image contains a set of 2Dimage data. Optionally, attaching the set of 2D image data to the 3Dpolygon prism model is achieved by performing a LIV mapping scheme toattach the image to the corresponding plane of the 3D polygon prismmodel. Here, the UV mapping refers to a 3D modeling process ofprojecting a 2D image to a 3D model's surface for texture mapping. For ahexagon prism model, the multiple sample sub-images include six samplesub-images corresponding to six horizontal side planes and two samplesub-images corresponding to two end planes (one top plane and one bottomplane). FIG. 4 is an exemplary diagram showing eight sample sub-imagessampled from either planes of a hexagon prism display model according toan embodiment of the present disclosure.

Referring to FIG. 4 , in horizontal projections there are six 2D imagesH1˜H6 mapped onto respective six side planes 101-1˜101-6 of a hexagonprism model 10 (see. FIG. 2 ) and two 2D images U and D mapped ontorespective a top plane 102 and a bottom plane 103 (see FIG. 2 ). Each 2Dimages sampled by respective sampling camera can be defined with a sameresolution or independently defined with different resolutions dependingon numbers of detailed textures in each individual 2D image. Forexample, in one projection direction, a scene has less number of objectswith less variation in color, the sub-image in this projection directioncan be sampled with a low resolution. In another example, in anotherprojection direction, a scene contains more objects with multiplecolors, the sub-image in this projection direction can be sampled withan enhanced resolution.

In some embodiments, the step of using multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images includes capturing at least two sample sub-images inprojected squares respectively for top view toward a top plane andbottom view toward a bottom plane by slightly enlarging field-of-viewangles to make the projected squares to be larger than the top plane orthe bottom plane. In a specific embodiment, each of the two 2D images Uand D sampled respectively for the top and bottom planes are two squareshaped, images sufficiently large (by enhancing the vertical FOV angleto cover the area of the two hexagons). Since a rendering pipeline ofrendering engine can only take rectangular matrix of data to performrendering of a 3D scene to a 2D screen, it is unable to capture othershaped images.

In an embodiment, as these 2D images are attached to respective planesof the 3D display model, e.g., the hexagon prism model, a constructeddisplay model is formed for the rendering engine. Optionally, eachsample sub-image of the eight 2D images is mapped to one correspondingplane of the 3D display panel through a UV mapping process. The UVmapping refers to a 3D modeling process of projecting a 2D image to a 3Dmodel's surface for texture mapping. In particular, the 2D imagesubstantially represents textures added to a corresponding surfaceregion of the 3D model. The constructed display model becomes anexecutable data form that is subjected to treatment by the renderingengine.

Referring to FIG. 1 , the method for generating a panorama image basedon a rendering engine includes a step of rendering the constructeddisplay model by the rendering engine to reconstruct a panorama image.Optionally, rendering the constructed display model to use allindividual sample sub-images collectively to construct a panorama image.FIG. 5 is a schematic diagram of a virtual panorama image generated froma constructed display model by running a rendering engine according toan embodiment of the present disclosure. On the left, a perspective viewof the constructed display model is shown with two rectangular shapedside planes 201-1 and 201-2 (attached with corresponding sub-imagesthough not shown) and a top plane 202 and a bottom plane 203 (withenlarged square shape). On the right, a portion of a virtual panoramaimage 500 is generated by the rendering engine with three sub-images H1,H2, and H3 being connected through the rendering process and partiallyindicated by the solid lines added therein for illustration purpose(which are otherwise not visible in the virtual panorama image 500.

In some embodiments, the step of using multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images includes using two sets of sampling cameras to capture twosets of sample sub-images with parallax for the multiple sub-planes; andreconstructing a 3D panorama image based on the two sets of samplesub-images with parallax. For example, the eight 2D images in FIG. 4 canbe one of two sets of eight 2D images separately captured by two sets ofsampling cameras that are disposed based on results of a certainparallax calculation. Thus, the two sets of eight 2D images are two setof images correlated with the certain parallax and can be used toconstruct a virtual panorama image with 3D visual effect by therendering engine.

In some embodiments, the step of using multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images further includes sampling a series of sample sub-imagestime-sequentially with a sampling frequency for each of the multiplesub-planes of the display model; encoding all sample sub-images of asame sub-plane according to an order of being sampled to generate asample sub-video; and attaching the sample sub-video to a correspondingsub-plane to form the constructed display model. For example, the eight2D images in FIG. 4 can be one of a series of sets of eight 2D imagescaptured by a set of sampling cameras sequentially in different timepoints with a fixed frequency such as 24 Hz or 30 Hz. In one example, 24sets or 30 sets of eight 2D images are collected per second. Each 2Dimage is a sample sub-image corresponding to a field of view in thescene. Optionally, each group of the sample sub-images belonging to asame field of view can be encoded according to their timings of beingcaptured. Optionally, these images can be encoded using video processingtools such as FFMPEG or OPENCV or similar software products to make onesub-video corresponding to the same field of view of the scene. Eachsub-video can be displayed virtually on the display model in terms ofrunning the rendering engine. As all sub-videos corresponding torespectively fields of views of the scene are displayed on the displaymodel, a virtual panorama video can be generated by the rendering engineand displayed on the virtual reality display apparatus. Optionally, thehigh-resolution panorama video can be displayed in such a way to allowindividual sub-video to be displayed on a lower resolution displayer(such as one based on Windows system) without installing a third-partydecoder to handle the high-resolution video.

FIG. 6 is an exemplary diagram showing partial visual effect images fromseveral random field-of-view angles out of the virtual panorama imagegenerated by the rendering engine according to some embodiments of thepresent disclosure. Referring to FIG. 6 , each partial visual effectimage, e.g., image 501, 502, or 503, represents a portion of the virtualpanorama image generated according to the present method. Each of thesevisual effect images show no substantial image distortion and patchaggregation.

FIG. 7 is a schematic diagram of a display model for the renderingengine for generating a panorama image according to another embodimentof the present disclosure. In another embodiment, the display model isstill a hexagon prism but with an upgraded sub-plane setup. Inparticular, the hexagon prism is divided equally into four layers alonga central axis direction so that each side plane are divided into foursub-planes, leading to total 26 sub-planes that need to add textures byattaching corresponding sample sub-images onto thereof. As shown in FIG.7 , the first layer includes N side sub-planes: 701-1-1˜701-1-N. TheM-th layer also includes N side sub-planes: 701-M-1˜701-M-N. In anexample, M=4 and N=6. Additionally, as any polygon prism model there aretwo end sub-planes, one top plane 702 and one bottom plane 703. In acondition that considering pixel uniformity, each side sub-plane can beprovided with a maximum image resolution of 8192×2928. With thehorizontal FOV angle being 60° and vertical FOV angle being 90°, thenumber of pixels within such FOV angles can be reached up to 8192×11712.This results in a final panorama image having an image resolution 8times higher than that based on the hexagon prism model 10 and 34 timeshigher than that based on conventional sphere model 00 shown in FIG. 2 .

In another aspect, the present disclosure provides an apparatus forgenerating a panorama image based on a rendering engine associated witha display apparatus. Optionally, the display apparatus is a virtualreality (VR) display apparatus configured to display high resolutionimage with resolution of at least 4K or 8K. In an embodiment, theapparatus includes a memory and one or more processors. The memory andthe one or more processors are connected with each other through anetwork connection. The network connection may be through acommunication network, such as a wireless network, a wired network,and/or any combination of a wireless network and a wired network. Thenetwork may include a local area network, the Internet, atelecommunications network (Internet of Things), and/or any combinationof the above-networks, etc. The wired network can communicate by meansof twisted pair, coaxial cable or optical fiber transmission. A wirelesscommunication network such as 3G/4G/5G mobile communication network,Bluetooth, Zigbee or Wi-Fi can be used. The memory storescomputer-executable instructions for controlling the one or moreprocessors. The memory may include static random-access memory (SRAM),electrically erasable programmable read-only memory (EEPROM), erasableprogrammable read-only memory (EPROM), read-only memory (ROM), magneticmemory, flash memory, disk, or optical disk. The memory storescomputer-executable instructions for controlling the one or moreprocessors to determine a display model configured to be a polygon prismfor a rendering engine corresponding to a panorama view of a scene.Additionally, the memory stores computer-executable instructions forcontrolling the one or more processors to use multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images of the panorama view in respective directions towardsmultiple sub-regions of the polygon prism. The memory storescomputer-executable instructions for controlling the one or moreprocessors further to attach the multiple sample sub-images torespective multiple sub-regions to form a constructed display model.Furthermore, the memory stores computer-executable instructions forcontrolling the one or more processors to render the constructed displaymodel by the rendering engine to reconstruct a panorama image.

In an embodiment, the one or more processors includes a programmableinstruction to create a display model configured as an equilateralpolygon prism with each of side, top, and bottom planes being dividedinto one or more sub-planes. Optionally, each sub-plane is independentlydefined with an image resolution. Optionally, each sub-plane isself-defined with an image resolution.

In an embodiment, the one or more processors include a rendering enginehaving multiple sampling cameras configured to capture multiple samplesub-images of a scene in multiple projection directions from at leastone of the multiple sampling cameras located inside the display modeltowards one of the respective multiple sub-planes.

In an embodiment, the one or more processors include a programmableinstruction to attach each of the multiple sample sub-images as texturesto respective one of the multiple sub-planes of the polygon prism toform a constructed display model for the rendering engine. The renderingengine is configured to render the multiple sample sub-images associatedwith the respective multiple sub-planes of the constructed display modelto generate a panorama image. The rendering engine is further configuredto sequentially render multiple sets of the multiple sample sub-imagesassociated with the respective multiple sub-planes of the constructeddisplay model. Each set of the multiple sample sub-images issequentially captured by a respective one of the multiple samplingcameras with a frequency of, e.g., 24 Hz or 30 Hz, and encoded in atiming order to generate respective multiple sample sub-videosassociated with the multiple sub-planes and generate a panorama video tobe displayed on the display apparatus.

In yet another aspect, the present disclosure provides a displayapparatus including a display panel coupled to the apparatus describedherein for generating a panorama image or panorama video. The displayapparatus is configured to display the panorama image or panorama videoas a virtual reality display. Optionally, the display apparatus is avirtual reality (VR) display apparatus. Optionally, the displayapparatus is built in lap top computer, desk top computer under Windowsor Apple OP systems, smart phone under Android or Apple mobile OPsystems, or any VR displayers.

In still another aspect, the present disclosure provides acomputer-program product including a non-transitory tangiblecomputer-readable medium having computer-readable instructions thereon.The computer-readable instructions are executable by a processor tocause the processor to perform determining a display model configured tobe a polygon prism for a rendering engine corresponding to a panoramaview of a scene. Additionally, the computer-readable instructions areexecutable by a processor to cause the processor to perform usingmultiple sampling cameras associated with the rendering engine tocapture multiple sample sub-images of the panorama view in respectivedirections towards multiple sub-regions of the polygon prism.Furthermore, the computer-readable instructions are executable by aprocessor to cause the processor to perform attaching the multiplesample sub-images to respective multiple sub-regions to form aconstructed display model. Moreover, the computer-readable instructionsare executable by a processor to cause the processor to performrendering the constructed display model by the rendering engine toreconstruct a panorama image. Optionally, the non-transitory tangiblecomputer-readable medium is stored in a display apparatus containing arendering engine. Optionally, the display apparatus is a virtual realitydisplay apparatus.

The foregoing description of the embodiments of the invention has beenpresented for purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formor to exemplary embodiments disclosed. Accordingly, the foregoingdescription should be regarded as illustrative rather than restrictive.Obviously, many modifications and variations will be apparent topractitioners skilled in this art. The embodiments are chosen anddescribed in order to explain the principles of the invention and itsbest mode practical application, thereby to enable persons skilled inthe art to understand the invention for various embodiments and withvarious modifications as are suited to the particular use orimplementation contemplated. It is intended that the scope of theinvention be defined by the claims appended hereto and their equivalentsin which all terms are meant in their broadest reasonable sense unlessotherwise indicated. Therefore, the term “the invention”. “the presentinvention” or the like does not necessarily limit the claim scope to aspecific embodiment, and the reference to exemplary embodiments of theinvention does not imply a limitation on the invention, and no suchlimitation is to be inferred. The invention is limited only by thespirit and scope of the appended claims. Moreover, these claims mayrefer to use “first”, “second”, etc. following with noun or element.Such terms should be understood as a nomenclature and should not beconstrued as giving the limitation on the number of the elementsmodified by such nomenclature unless specific number has been given. Anyadvantages and benefits described may not apply to all embodiments ofthe invention. It should be appreciated that variations may be made inthe embodiments described by persons skilled in the art withoutdeparting from the scope of the present invention as defined by thefollowing claims. Moreover, no element and component in the presentdisclosure is intended to be dedicated to the public regardless ofwhether the element or component is explicitly recited in the followingclaims.

What is claimed is:
 1. A method for generating a panorama image based ona rendering engine associated with a display apparatus comprising:determining a display model configured to be a polygon prism for therendering engine corresponding to a panorama view of a scene; usingmultiple sampling cameras associated with the rendering engine tocapture multiple sample sub-images of the panorama view in respectivedirections towards multiple sub-planes of the polygon prism; attachingthe multiple sample sub-images to respective multiple sub-planes to forma constructed display model; rendering the constructed display model bythe rendering engine to reconstruct a panorama image; and displaying thepanorama image on the display apparatus; wherein the polygon prism is anequilateral N-side polygon prism; and determining the display modelcomprises determining different image resolutions corresponding torespective multiple sub-planes of different planes of the equilateralN-side polygon prism and determining a width and a length as well as awidth-to-length ratio of each of the multiple sub-planes based on aratio of tan(u/2)/tan(v/2), where u is a horizontal field-of-view angleand v is a vertical field-of-view angle projected from at least one ofthe multiple sampling cameras to each of the multiple sub-planes.
 2. Themethod of claim 1, wherein N is an integer no smaller than
 3. 3. Themethod of claim 2, wherein determining the display model furthercomprises determining the multiple sub-planes divided respectively fromeach side or top or bottom plane of the equilateral N-side polygonprism.
 4. The method of claim 1, wherein the at least one of themultiple sampling cameras is located at a center of the display model.5. The method of claim 1, wherein using multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images comprises separately sampling each of the multiple samplesub-images with an independently-defined image resolution depending on ascene in respective one direction toward one of multiple sub-planesprojected from one of the multiple sampling cameras.
 6. The method ofclaim 5, wherein the independently-defined image resolution for a singlesub-plane is configured to be several times greater than a maximum imageresolution allowed for a single field-of-view image in a conventionalmodel.
 7. The method of claim 1, wherein using multiple sampling camerasassociated with the rendering engine to capture multiple samplesub-images comprises capturing at least two sample sub-images inprojected squares respectively for top view toward a top plane andbottom view toward a bottom plane by slightly enlarging field-of-viewangles to make the projected squares to be larger than the top plane orthe bottom plane.
 8. The method of claim 1, wherein using multiplesampling cameras associated with the rendering engine to capturemultiple sample sub-images comprises using two sets of sampling camerasto capture two sets of sample sub-images with parallax for the multiplesub-planes; and reconstructing a 3D panorama image based on the two setsof sample sub-images with parallax.
 9. The method of claim 1, whereinattaching the multiple sample sub-images to respective multiplesub-planes comprises performing UV mapping to add image textures of eachsample sub-image to a corresponding sub-plane and generating theconstructed display model for the rendering engine.
 10. The method ofclaim 1, wherein using multiple sampling cameras associated with therendering engine to capture multiple sample sub-images further comprisessampling a series of sample sub-images time-sequentially with a samplingfrequency for each of the multiple sub-planes of the display model;encoding all sample sub-images of a same sub-plane according to an orderof being sampled to generate a sample sub-video; and attaching thesample sub-video to a corresponding sub-plane to form the constructeddisplay model.
 11. The method of claim 10, wherein rendering theconstructed display model comprises rendering multiple sample sub-imagesrespectively for the multiple sub-planes that are sampled at a same timeto generate one panorama image at the same time and further generating apanorama video by encoding a series of panorama images sequentially intime.
 12. A method for generating a panorama image based on a renderingengine associated with a display apparatus comprising: determining adisplay model configured to be a polygon prism for the rendering enginecorresponding to a panorama view of a scene; using multiple samplingcameras associated with the rendering engine to capture multiple samplesub-images of the panorama view in respective directions towardsmultiple sub-planes of the polygon prism; attaching the multiple samplesub-images to respective multiple sub-planes to form a constructeddisplay model; rendering the constructed display model by the renderingengine to reconstruct a panorama image; and displaying the panoramaimage on the display apparatus; wherein rendering the constructeddisplay model comprises rendering multiple sample sub-imagesrespectively for the multiple sub-planes that are sampled at a same timeto generate one panorama image at the same time and further generating apanorama video by encoding a series of panorama images sequentially intime; and displaying the panorama image comprises displaying thepanorama video on the display apparatus by at least displaying asub-video separately for respective one sub-plane in a direction of afield-of-view.
 13. An apparatus for generating a panorama image based ona rendering engine associated with a display apparatus, comprising: amemory; and one or more processors; wherein the memory and the one ormore processors are connected with each other; and the memory storescomputer-executable instructions for controlling the one or moreprocessors to: determine a display model configured to be a polygonprism for the rendering engine corresponding to a panorama view of ascene; use multiple sampling cameras associated with the renderingengine to capture multiple sample sub-images of the panorama view inrespective directions towards multiple sub-planes of the polygon prism;attach the multiple sample sub-images to respective multiple sub-planesto form a constructed display model; render the constructed displaymodel by the rendering engine to reconstruct a panorama image; anddisplay the panorama image on the display apparatus; wherein the polygonprism is an equilateral N-side polygon prism; and the memory furtherstores computer-executable instructions for controlling the one or moreprocessors to determine different image resolutions corresponding torespective multiple sub-planes of different planes of the equilateralN-side polygon prism and determining a width and a length as well as awidth-to-length ratio of each of the multiple sub-planes based on aratio of tan(u/2)/tan(v/2), where u is a horizontal field-of-view angleand v is a vertical field-of-view angle projected from at least one ofthe multiple sampling cameras to each of the multiple sub-planes. 14.The apparatus of claim 13, wherein the one or more processors iscontrolled by a programmable instruction to create a display modelconfigured as an equilateral polygon prism with each of side, top, andbottom planes being divided into one or more sub-planes, each sub-planebeing independently defined with an image resolution.
 15. The apparatusof claim 14, wherein the one or more processors comprises a renderingengine having multiple sampling cameras configured to capture multiplesample sub-images of a scene in multiple projection directions from atleast one of the multiple sampling cameras located inside the displaymodel towards one of the respective multiple sub-planes.
 16. Theapparatus of claim 15, wherein the one or more processors is furthercontrolled by a programmable instruction to attach each of the multiplesample sub-images as textures to respective one of the multiplesub-planes of the polygon prism to form a constructed display model forthe rendering engine; wherein the rendering engine is configured torender the multiple sample sub-images associated with the respectivemultiple sub-planes of the constructed display model to generate apanorama image.
 17. The apparatus of claim 16, wherein the renderingengine is configured to sequentially render multiple sets of themultiple sample sub-images associated with the respective multiplesub-planes of the constructed display model, each set of the multiplesample sub-images being sequentially captured by a respective one of themultiple sampling cameras with a sampling frequency and encoded in atiming order to generate respective multiple sample sub-videosassociated with the multiple sub-planes, and to generate a panoramavideo.
 18. A display apparatus comprises a display panel coupled to theapparatus according to claim 13 for generating a panorama image orpanorama video and being configured to display a panorama image orpanorama video as a virtual reality display.
 19. A computer-programproduct comprising a non-transitory tangible computer-readable mediumhaving computer-readable instructions thereon, the computer-readableinstructions being executable by a processor to cause the processor toperform: determining a display model configured to be a polygon prismfor a rendering engine corresponding to a panorama view of a scene;using multiple sampling cameras associated with the rendering engine tocapture multiple sample sub-images of the panorama view in respectivedirections towards multiple sub-planes of the polygon prism; attachingthe multiple sample sub-images to respective multiple sub-planes to forma constructed display model; and rendering the constructed display modelby the rendering engine to reconstruct a panorama image; wherein thepolygon prism is an equilateral N-side polygon prism; and thecomputer-readable instructions are further executable by a processor tocause the processor to perform determining different image resolutionscorresponding to respective multiple sub-planes of different planes ofthe equilateral N-side polygon prism and determining a width and alength as well as a width-to-length ratio of each of the multiplesub-planes based on a ratio of tan(u/2)/tan(v/2), where u is ahorizontal field-of-view angle and v is a vertical field-of-view angleprojected from at least one of the multiple sampling cameras to each ofthe multiple sub-planes.